Augmented reality device with predefined object data

ABSTRACT

Techniques for displaying an augmented reality toy. Embodiments capture a visual scene for display. The visual scene includes a first physical object and is captured using one or more camera devices. The first physical object is identified as a first predetermined object type, based on one or more object identifiers associated with the first physical object. Embodiments retrieve predefined geometric information corresponding to the first predetermined object type and render a sequence of frames for display in which the captured visual scene is augmented, based on the predefined geometric information.

BACKGROUND

Field of the Invention

The present invention generally relates to a human-computer interfaceand more specifically to techniques for recognizing and displayingpredefined objects on an augmented reality device.

Description of the Related Art

Computer graphics technology has come a long way since video games werefirst developed. Relatively inexpensive 3D graphics engines now providenearly photo-realistic interactive game play on hand-held video game,home video game and personal computer hardware platforms costing only afew hundred dollars. These video game systems typically include ahand-held controller, game controller, or, in the case of a hand-heldvideo game platform, an integrated controller. A user or player uses thecontroller to send commands or other instructions to the video gamesystem to control a video game or other simulation being played. Forexample, the controller may be provided with a manipulator (e.g., ajoystick) and buttons operated by the user.

Many hand-held gaming devices include some form of camera device whichmay be used to capture an image or a series of images of a physical,real-world scene. The captured images can then be displayed, forinstance, on a display of the hand-held gaming device. Certain devicesmay be configured to insert virtual objects into the captured imagesbefore the images are displayed. Additionally, other devices orapplications may enable users to draw or paint particular within acaptured image of a physical scene. However, as such alterations applyonly to a single image of the physical scene, subsequent captured imagesof the physical scene from different perspectives may not incorporatethe user's alterations.

SUMMARY

Embodiments provide a method, computer-readable memory and augmentedreality device for displaying a first physical object. The method,computer-readable memory and augmented reality device include capturinga visual scene for display. The visual scene includes the first physicalobject and is captured using one or more camera devices. The method,computer-readable memory and augmented reality device also includeidentifying the first physical object as a first predetermined objecttype, based on one or more object identifiers associated with the firstphysical object. Additionally, the method, computer-readable memory andaugmented reality device include retrieving predefined geometricinformation corresponding to the first predetermined object type. Themethod, computer-readable memory and augmented reality device furtherinclude rendering a sequence of frames for display in which the capturedvisual scene is augmented, based on the predefined geometric information

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited aspects are attained andcan be understood in detail, a more particular description ofembodiments of the invention, briefly summarized above, may be had byreference to the appended drawings.

It is to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

FIG. 1 is a block diagram illustrating an augmented reality deviceconfigured with an augmented reality component, according to oneembodiment described herein.

FIG. 2 illustrates an augmented reality device viewing an augmentedreality toy, according to one embodiment described herein.

FIG. 3 is a screenshot of the screen of the augmented reality deviceshown in FIG. 2, according to one embodiment described herein.

FIG. 4 is a flow diagram illustrating a method for displaying anaugmented reality toy on an augmented reality device, according to oneembodiment described herein.

FIG. 5 is a block diagram illustrating an augmented reality deviceconfigured with an augmented reality component, according to oneembodiment described herein.

DETAILED DESCRIPTION

Generally, embodiments of the invention provide techniques fordisplaying content on an augmented reality device. As used herein, anaugmented reality device refers to any device capable of displaying areal-time view of a physical, real-world environment while alteringelements within the displayed view of the environment. As such, unlike avirtual reality device which displays a view of virtual world, anaugmented reality device displays a view of the real world but augmentselements using computer graphics technology. Such an augmented realitydevice may include and/or be communicatively coupled to a camera device(or multiple camera devices) used to capture a view of the real-worldenvironment and may further include computer software and/or hardwareconfigured to augment elements of the captured scene. For example, anaugmented reality device could capture a series of images of a coffeecup sitting on top of a table, modify the series of images so that thecoffee cup appears as an animated cartoon character and display themodified series of images in real-time to a user. As such, when the userlooks at the augmented reality device, the user sees an augmented viewof the physical real-world environment in which the user is located.

Embodiments provide techniques for displaying an augmented reality toyon an augmented reality device. Software on the augmented reality devicemay capturing a visual scene for display using one or more cameras ofthe augmented reality device. The visual scene includes the augmentedreality toy. For example, cameras could be used to capture one or moreimages of a toy castle sitting atop a table. The software could identifythe augmented reality toy as a first predetermined object type, based onone or more object identifiers associated with the first physicalobject. For example, the toy castle could include a marker thatcorresponds to a particular object type, where different types ofaugmented reality toys are labeled with different markers, eachcorresponding to a respective object type. Additionally, the markercould be embedded using a material that is difficult or impossible tosee with the human eye (e.g., an infrared-absorbing ink). In such anexample, the augmented reality device could be configured with aninfrared camera capable of detecting the embedded marker and, upondetecting the embedded marker, the software could determine thepredetermined object type that the particular marker corresponds to(e.g., a particular type of toy castle).

Furthermore, the software could be configured to identify the objecttype of the augmented reality toy based on its shape. For instance, thesoftware on the augmented reality device could analyze the visual sceneto determine a plurality of edges of the first physical object withinthe visual scene, and could use the determined plurality of edges toidentify the predetermined object type of the augmented reality toy. Inone embodiment, the augmented reality toy is configured with atransmitter (e.g., a RF transmitter) that transmits a signal withembedded data specifying an object type identification code. Software onthe augmented reality device could receive the signal and coulddetermine the predetermined object type based on the specified objecttype identification code.

Additionally, the software could retrieve predefined geometricinformation corresponding to the first predetermined object type. Thegeometric information could specify, for instance, dimensions of objectsin the predetermined object type, the shape of the objects in thepredetermined object type, and so on. Additionally, the geometricinformation could identify one of more effect areas on the objects inthe predetermined object type. Continuing the example, the toy castlecould include several windows and these could be identified as effectareas in the geometric data for the castle.

The software could then render a sequence of frames for display in whichan appearance of the first physical object is augmented, based on thepredefined geometric information. For example, the window effect areason the toy castle could be augmented to appear as if light is emittingfrom the windows. As another example, one or more animated virtualcharacters could be depicted on, in or around the augmented reality toy.For instance, an animated virtual character could be shown through thecastle windows in the sequence of frames walking around a room withinthe toy castle. Of note, such a depiction may be shown, even though thetoy castle itself may not include any interior rooms. For example, whenviewed outside of the augmented reality device, the toy castle could bea plastic castle with several stickers depicting windows. However, whenviewed through the augmented reality device, these windows could appearas realistic windows emitting light from an interior room of the castle,and with one or more animated virtual characters moving throughout theroom. Advantageously, doing so provides an improved experience for usersof the augmented reality toy.

FIG. 1 is a block diagram illustrating an augmented reality deviceconfigured with a display correction component, according to oneembodiment of the present invention. As shown, the augmented realitydevice 100 includes an augmented reality component 110, camera devices120, a display device 130 and an accelerometer 140. The camera devices120 may include cameras for capturing a visual scene. As used herein, avisual scene refers to a view(s) of the real-world environment in whichthe device 100 is being used. For instance, a visual scene may be aseries of images of a real-world environment. The camera devices 120 mayalso include one or more user-facing cameras. The augmented realitycomponent 110 could use such a user-facing camera device 120 to, e.g.,determine an angle at which the user is viewing the display device 130.Generally, the accelerometer 140 is a device capable of measuring thephysical (or proper) acceleration of the augmented reality device 100.The augmented reality component 110 may use the accelerometer 140 to,e.g., determine when the position of the augmented reality device 100 ischanging, which could indicate the user's viewing angle of the displaydevice 130 is also changing.

Generally, the augmented reality component 110 is configured torecognize augmented reality toys within a visual scene (e.g., a seriesof frames captured using the camera devices 120) and to adjust thedepiction of the visual scene on the augmented reality device based onpredefined data associated with the augmented reality toys. Forinstance, the augmented reality component 110 could analyze a visualscene captured using the cameras 120 and identify augmented reality toyswithin the visual scene. More specifically, as the visual scenerepresents a three-dimensional space (i.e., the physical environmentcaptured using the cameras 120), the augmented reality component 110could determine an area of three-dimensional space occupied by eachidentified augmented reality toy. For example, the augmented realitycomponent 110 could be preconfigured with geometric data that definesgeometric properties (e.g., size, shape, color, etc.) for particulartoys, and could use the geometric data to identify instances of thepredefined toys within the visual scene and the three-dimensional spaceeach object occupies.

In one embodiment, the augmented reality toy is configured with atransmitter (e.g., a radio frequency (RF) transmitter) that sends out asignal encoded with data specifying a type identifier. In such anembodiment, the augmented reality component 110 could receive the signal(e.g., using a receiver or transceiver on the augmented reality device100) and could determine the type identifier encoded within the signal.The augmented reality component 110 could then determine the type of thetoy, based on the type identifier.

In another embodiment, the augmented reality toy may contain an embeddedmarker that identifies the type of the toy. For instance, the augmentedreality toy could contain a quick response (QR) code that specifies atype identifier corresponding to the type of the toy. More generally,however, any type of marker capable of identifying a type or a typeidentifier may be used. The augmented reality component 110 could thendetect the embedded marker (e.g., using a camera 120 of the augmentedreality device 100) and could determine the type of the toy based on theembedded marker. In a particular embodiment, the embedded marker isimpossible or difficult to detect using the human eye. For example, theembedded marker could be expressed using an infrared-absorbing materialthat is invisible or nearly invisible to the human eye, and theaugmented reality component 110 could be configured to detect theembedded marker using one or more infrared cameras on the augmentedreality device 100. Advantageously, doing so allows the marker to beembedded in the augmented reality toy without disrupting the aestheticsof the toy.

Upon identifying an augmented reality toy within the visual scene, theaugmented reality component 110 could then retrieve predefined dataassociated with the identified toy. For example, the augmented realitycomponent 110 could determine that the augmented reality toy is a castleand could retrieve augmentation data associated with the castle objecttype. Such augmentation data could specify, for instance, one or moreareas of the castle toy to augment and how the areas should beaugmented. As an example, the physical castle toy could include severalstickers that depict windows of the castle, the augmentation data couldspecify that these stickers should be augmented to appear as realwindows that emit light. Additionally, the augmentation data couldspecify that the augmented windows should depict one or more animatedvirtual characters shown within the castle. The augmented realitycomponent 110 could then render a series of frames depicting anaugmented virtual scene based on the augmentation data. Advantageously,by recognizing the physical toy as a particular type of augmentedreality toy, the augmented reality component 110 can provideaugmentations that are specific to the particular type of toy, therebyenhancing the appearance of the toy and the user's experience with thetoy.

Additionally, the augmented reality component 110 could depictinteractions between virtual characters and the augmented reality toybased on the type of the toy. For instance, upon detecting an arcticcastle toy, the augmented reality component 110 could generate a seriesof frames depicting an ice patch next to the toy. Moreover, upondetermining that a virtual character within the augmented reality sceneis coming into contact with the ice patch, the augmented realitycomponent 110 could depict the virtual character as slipping on the ice.Advantageously, doing so helps to create a more immersive and improvedexperience for users of the augmented reality toy.

In addition to identifying the type of the augmented reality toy, theaugmented reality component 110 can use predefined geometric dataassociated with the type of toy to augment the augmented reality toy'sappearance. For instance, such geometric data could specify the shapeand dimensions of a staircase on the augmented reality castle, and theaugmented reality component 110 could use this information to renderframes realistically depicting a virtual character walking up the stepsof the toy castle. Additionally, by pre-configuring the augmentedreality component 110 with geometric data specifying the shape of thestairs, the augmented reality component 110 does not need to approximatethe shape and size of the stairs based on the toy's appearance in thecaptured visual scene.

Additionally, the augmented reality component 110 on the augmentedreality device could measure one or more environmental illuminationcharacteristics of an environment in which the augmented reality deviceis located. Environmental illumination characteristics could include,for instance, a position of a light source within an environment inwhich the augmented reality device is located, an angle of the lightsource, an indication of whether the light source is omnidirectional, acolor of the light source, and an intensity of the light source and areflectivity value of the first physical object. Based on theenvironmental illumination characteristics, the augmented realitycomponent 110 could adjust the appearance of the augmented firstphysical object and virtual characters/objects within the augmentedreality scene, based on the measured one or more environmentalillumination characteristics. For instance, the augmented realitycomponent 110 could identify one or more shadows within the visual sceneand could render shadows for one or more virtual characters or objectswithin the augmented reality scene based on the identified shadows. Asan example, the augmented reality component 110 could determine that atoy castle has a shadow on the left side of the captured visual scene,indicating that a light source is shining on the toy castle from theright side of the captured visual scene. In such an example, theaugmented reality component 110 could render shadows for virtual objectsand characters in the augmented reality scene, based on a virtual lightsource shining from the right side of the augmented reality scene.

While the aforementioned examples refer to identifying light sourcesbased on shadows of physical objects within the captured visual scene,these examples are without limitation and it is contemplated thatnumerous other techniques could be used to identify light sources withinthe physical environment. For instance, the augmented reality device 100could be configured with multiple cameras positioned on multiple,different sides of the device 100, and the augmented reality component110 could use images from these other cameras to identify light sourcespositioned throughout the physical environment. As another example, therendered sequence of frames could depict a virtual pond positioned onthe table next to the toy castle and could augment the appearance of thevirtual pond to show reflections from one or more light sources withinthe environment. Moreover, the augmented reality component 110 coulddepict these reflections as having an effect on other virtualobjects/characters or the physical toy within the augmented realityscene. For instance, the augmented reality component 110 could depictlight reflected from the virtual pond shining onto the walls of the toycastle. Doing so provides a more dynamic and realistic augmented realityworld that is capable of adapting to the environment in which theaugmented reality device is located.

In addition to augmenting the appearance of the augmented reality toy,the augmented reality component 110 could also augment the acoustics ofthe toy. For instance, the augmented reality component 110 could beconfigured to recognize a stuffed animal dog toy, and when viewing thetoy dog with the augmented reality device, the augmented realitycomponent 110 could play sound effects associated with the toy dog. Forinstance, when the user views the toy dog with the augmented realitydevice, the augmented reality component 110 could render a series offrames depicting the toy dog as an animated dog and could further playsound effects corresponding to the animation (e.g., a barking noise whenthe animated dog barks).

Additionally, the augmented reality component 110 could be configured todepict interactions between animated virtual character and the augmentedreality toy based on a set of dynamics rules. The dynamics rules maydefine dynamics interactions for visual scenes displayed on theaugmented reality device. In one embodiment, the dynamics rules used maybe determined based on the type of augmented reality toy in the visualscene. As an example, a spaceship augmented reality toy could beassociated with a set of low-gravity dynamics rules and the augmentedreality component 110, upon detecting the visual scene includes thespaceship toy, could apply the set of low-gravity dynamics rules tovirtual characters within the augmented reality scene.

While the aforementioned example describes an embodiment configured toaugment a three-dimensional toy's appearance, such an example is withoutlimitation and is provided for illustrative purposes only. Moreover, itis explicitly contemplated that embodiments can be configured tointeract with two-dimensional objects as well. For example, an augmentedreality device could be configured to recognize and augment images shownon the pages of a story book. As an example, a first page of a storybook could include a picture of a castle and could include an embeddedmarker (e.g., a unique symbol embedded in the page using an infraredink). In such an example, the augmented reality component 110 couldcapture a visual scene including the page of the book and could furtherdetect the embedded marker (e.g., using an infrared camera). Theaugmented reality component 110 could then render frames depicting thecastle on the page as having one or more augmentations. For example, theaugmented castle could appear to stand out from the page and have athree-dimensional appearance. Additionally, a virtual character could beshown moving about the page and interacting with the castle.Advantageously, doing so allows the two-dimensional picture of thecastle to, in effect, “come alive” with an altered appearance and/orinteractions with virtual characters in the augmented reality world,thereby enhancing the user's experience with the story book.

FIG. 2 illustrates an augmented reality device viewing an augmentedreality toy, according to one embodiment described herein. As shown, thescene 240 includes a toy castle 210 sitting atop a table 215.Additionally, the scene 240 includes an augmented reality device 100that is viewing the toy castle 210 and is rendering and displaying oneor more frames depicting an augmented reality scene on its displaydevice 245. As discussed above, the augmented reality component 110could identify the type of the toy castle 210 (e.g., based on theappearance of the toy castle, based on a type identifier encoded in asignal, based on an embedded marker, etc.) and could augment theappearance of the toy castle 210 as shown on the augmented realitydevice 100 based on the determined type.

As shown on the display device 245, a number of different augmentationshave been applied to the toy castle. FIG. 3 shows a screenshot of thescreen of the augmented reality device shown in FIG. 2, according to oneembodiment described herein. Here, the screenshot 300 includes a visualdepiction 310 of the castle 210 and a number of different augmentations.The augmentations include a moat 320, fireworks 330, ponies 340, a tree350 and a drawbridge 360. Of note, the physical toy castle 210 depictedin FIG. 2 does not include any of the augmentations 320, 330, 340, 350and 360, but instead these augmentations have been created and appliedto the castle's appearance on the augmented reality device 100 based ona determination of the toy type of the physical castle 210.

Moreover, the various augmentations may be static virtual objects oranimated virtual objects. For instance, in the depicted example, thedrawbridge 360 could appear as static, while the fireworks 330 couldappear as an animated virtual object. Additionally, the various virtualobjects depicted in the augmented reality scene may appear to interactwith one another. For instance, the ponies 340 could appear to walkaround the augmented reality scene and could enter the castle bycrossing the drawbridge 360. Furthermore, in some situations, thevirtual objects 340 may appear as fully or partially occluded by othervirtual objects or by the toy castle 310. For example, as the poniescross the draw bridge, they could be partially or fully occluded by thecastle 310. In one embodiment, the augmented reality component 110 isconfigured to optimize the depicted scene by performing occlusionculling operations for one or more of the virtual objects.

In one embodiment, the virtual objects in the augmented reality sceneare depicted as visually affecting other objects (both virtual andphysical) within the scene. For instance, as the fireworks 330 explode,the augmented reality component 110 could augment the appearance of thecastle 310 (i.e., the physical toy) so that it appears light from theexploding fireworks is reflecting on the castle 310. Additionally, theaugmented reality component 110 could augment the appearance of thewater 320 (i.e., a virtual object) could be augmented so show thereflection of the exploding fireworks.

As discussed above, the augmented reality component 110 is configured todetermine an object type of the toy castle and to generate theaugmentations based on the determined type. Thus, while the particularcastle includes augmentations such as fireworks 330 and a moat 320, adifferent toy castle (e.g., an ice castle) could include other,different augmentations (e.g., Eskimos, polar bears, etc.). Moregenerally, it is broadly contemplated that any type of augmented realitytoy and virtual objects may be used, consistent with the functionalitydescribed herein. Advantageously, by determining an object type of theaugmented reality toy and by generating the augmented reality scenebased on the determined object type, embodiments can realisticallydepict augmentations for the augmented reality toy and can tailor theaugmentations to be contextually relevant to the augmented reality toy.

In one embodiment, the augmented reality component 110 is configured torender virtual characters that interact in different ways with thephysical toy, based on a state of the physical toy. For instance, assumethat a second castle toy includes a physical drawbridge that a child canopen and close. When the second castle toy is viewed with the augmentedreality device, the augmented reality component 110 could determinewhich state the drawbridge is currently in and could render animatedvirtual characters accordingly. Thus, for example, an animated virtualcharacter could appear to walk across the drawbridge and enter thecastle when the drawbridge is lowered (i.e., a first state), and theanimated virtual character could appear to be trapped either inside oroutside of the castle when the drawbridge is raised (i.e., a secondstate). Advantageously, doing so provides a more immersive andinteractive experience for users of the augmented reality device.

In a particular embodiment, the augmented reality component 110 on afirst augmented reality device is configured to synchronize with asecond augmented reality component 110 on a second augmented realitydevice. Such synchronization can occur between local augmented realitycomponents, remote augmented reality components, or a combinationtherebetween. For example, two separate users with two separateaugmented reality devices could view the same toy castle at the sametime. In such an example, the augmented reality components 110 on eachof the augmented reality devices could synchronize with one another,such that each of the two users sees the same augmentations to thecastle occurring at the same time. As another example, a remote usercould be viewing a separate instance of the toy castle remotely from thetwo users, but could be in contact (e.g., via video conferencing) withthe two users. In such an example, the augmented reality component 110on the remote user's augmented reality device could synchronize (e.g.,over a network such as the Internet) with the augmented reality devicesof the two local users, such that the two local users and the remoteuser all see the same augmentations occurring at the same time on theiraugmented reality devices. Advantageously, doing so helps to provide amore immersive experience for users of the augmented reality device,when there are multiple users viewing the same physical (and/ordifferent instances of the physical object) at the same time usingaugmented reality devices.

FIG. 4 is a flow diagram illustrating a method for displaying anaugmented reality toy on an augmented reality device, according to oneembodiment described herein. As shown, the method 400 begins at block410, where the augmented reality component 110 captures a visual scene.For instance, the augmented reality component 110 could use one or morecameras 120 on the augmented reality device 100 to capture the visualscene.

The augmented reality component 110 then identifies an augmented realitytoy within the visual scene and determines an object type of theaugmented reality toy (block 415). As discussed above, the augmentedreality component 110 could identify the augmented reality toy using avariety of techniques. For instance, the augmented reality component 110could be preconfigured with geometric data (e.g., size, shape,coloration, etc.) for various types of augmented reality toys, and theaugmented reality component 110 could use the geometric data to identifythe augmented reality toy within the visual scene as a particular objecttype. As an example, where the augmented reality component 110 isconfigured with the geometric data for several different types ofaugmented reality toys, the augmented reality component 110 coulddetermine which set of geometric data best matches the toy in the visualscene.

In one embodiment, the augmented reality component 110 is configured toidentify the augmented toy by detecting a marker embedded within thetoy. For example, a QR code could be embedded in the toy using aninfrared material (e.g., an infrared ink), such that the QR code isdifficult if not impossible to see with the human eye. In such anexample, the augmented reality component 110 could use one or moreinfrared cameras on the augmented reality device to detect the QR codeand could then determine an object type corresponding to the detected QRcode.

In a particular embodiment, the augmented reality toy is configured withan RF transmitter (or transceiver) that transmits a signal encoded withan object type identifier. The augmented reality component 110 couldthen use an RF receiver (or transceiver) in the augmented reality device100 to receive the signal. The augmented reality component 110 couldthen analyze determine the object type identifier encoded within thereceived signal and could determine the object type based on thisidentifier. Generally, any of the aforementioned techniques may be usedfor identifying the augmented reality toy, or a combination of thesetechniques may be used. For instance, the augmented reality component110 could identify the toy as a first object type based on the geometricdata and could confirm the identification by verifying that a signalspecifying the first object type is being broadcast.

Upon identifying the augmented reality toy, the augmented realitycomponent 110 augmented the appearance of the augmented reality toywithin the visual scene displayed on the augmented reality device (block420). For instance, the augmented reality component 110 could determineone or more augmentations associated with the determined object type ofthe augmented reality toy, and could render one or more frames depictingthe determined augmentation(s) applied to the captured visual scene. Asan example, where the augmented reality toy is a toy castle, theaugmented reality component 110 could determine that this toy isassociated with a fireworks augmentation. The augmented realitycomponent 110 could then render frames depicting virtual fireworks goingoff above the toy castle. Additionally, the rendered frames may augmentthe appearance of the toy castle as well based on the appliedaugmentations. For instance, the toy castle's appearance could beaugmented so that it appears light from the virtual fireworks isreflecting on to the castle displayed on the augmented reality device100. Once augmented reality component 110 renders frames depicting oneor more augmentations to the visual scene, the frames are output fordisplay (block 425) and the method 400 ends.

Additionally, the augmented reality component 110 could be configured touse the predefined geometric data for the augmented reality toy ingenerating the augmentations. For instance, assume that the toy castleincludes a flight of stairs leading up to the walls of the castle, andthat the predefined geometric data specifies the shape and dimensions ofthese stairs. The augmented reality component 110 could then use thepredefined geometric data to depict a virtual character walking up theflight of stairs. By preconfiguring the augmented reality device 100with data specifying the size and shape of the stairs, the augmentedreality component 110 can more accurately and realistically renderframes depicting a virtual character walking up the stairs. Doing soenhances the overall appearance of the rendered frames and thus mayimprove the user's experience with the augmented reality device and theaugmented reality toy as well.

FIG. 5 is a block diagram illustrating an augmented reality deviceconfigured with a surface painting component, according to oneembodiment described herein. In this example, the augmented realitydevice 100 includes, without limitation, a processor 500, storage 505,memory 510, I/O devices 520, a network interface 525, camera devices120, a display devices 130 and an accelerometer device 140. Generally,the processor 500 retrieves and executes programming instructions storedin the memory 510. Processor 500 is included to be representative of asingle CPU, multiple CPUs, a single CPU having multiple processingcores, GPUs having multiple execution paths, and the like. The memory510 is generally included to be representative of a random accessmemory. The network interface 525 enables the augmented reality device100 to connect to a data communications network (e.g., wired Ethernetconnection or an 802.11 wireless network). Further, while the depictedembodiment illustrates the components of a particular augmented realitydevice 100, one of ordinary skill in the art will recognize thataugmented reality devices may use a variety of different hardwarearchitectures. Moreover, it is explicitly contemplated that embodimentsof the invention may be implemented using any device or computer systemcapable of performing the functions described herein.

The memory 510 represents any memory sufficiently large to hold thenecessary programs and data structures. Memory 510 could be one or acombination of memory devices, including Random Access Memory,nonvolatile or backup memory (e.g., programmable or Flash memories,read-only memories, etc.). In addition, memory 510 and storage 505 maybe considered to include memory physically located elsewhere; forexample, on another computer communicatively coupled to the augmentedreality device 100. Illustratively, the memory 510 includes an augmentedreality component 110 and an operating system 515. The operating system515 generally controls the execution of application programs on theaugmented reality device 100. Examples of operating system 515 includeUNIX, a version of the Microsoft Windows® operating system, anddistributions of the Linux® operating system. (Note: Linux is atrademark of Linus Torvalds in the United States and other countries.)Additional examples of operating system 515 include custom operatingsystems for gaming consoles, including the custom operating systems forsystems such as the Nintendo DS® and Sony PSP®.

The I/O devices 520 represent a wide variety of input and outputdevices, including displays, keyboards, touch screens, and so on. Forinstance, the I/O devices 520 may include a display device used toprovide a user interface. As an example, the display may provide a touchsensitive surface allowing the user to select different applications andoptions within an application (e.g., to select an instance of digitalmedia content to view). Additionally, the I/O devices 520 may include aset of buttons, switches or other physical device mechanisms forcontrolling the augmented reality device 100. For example, the I/Odevices 520 could include a set of directional buttons used to controlaspects of a video game played using the augmented reality device 100.

The augmented reality component 110 generally is configured to renderframes for display on the augmented reality device that depict anaugmented reality toy. The augmented reality component 110 could capturea visual scene for display. Here, the visual scene could include a firstphysical object captured using the camera devices 120. The augmentedreality component 110 could identify the first physical object as afirst predetermined object type, based on one or more object identifiersassociated with the first physical object. Examples of such identifiersmay include an embedded marker within the first physical object, asignal received from a transmitter associated with the first physicalobject, and so on. The augmented reality component 110 may also retrievepredefined geometric information corresponding to the firstpredetermined object type. The augmented reality component 110 may thenrender a sequence of frames for display in which an appearance of thefirst physical object is augmented, based on the predefined geometricinformation.

In the preceding, reference is made to embodiments of the invention.However, the invention is not limited to specific described embodiments.Instead, any combination of the following features and elements, whetherrelated to different embodiments or not, is contemplated to implementand practice the invention. Furthermore, although embodiments of theinvention may achieve advantages over other possible solutions and/orover the prior art, whether or not a particular advantage is achieved bya given embodiment is not limiting of the invention. Thus, the precedingaspects, features, embodiments and advantages are merely illustrativeand are not considered elements or limitations of the appended claimsexcept where explicitly recited in a claim(s). Likewise, reference to“the invention” shall not be construed as a generalization of anyinventive subject matter disclosed herein and shall not be considered tobe an element or limitation of the appended claims except whereexplicitly recited in a claim(s).

Aspects of the present invention may be embodied as a system, method orcomputer program product. Accordingly, aspects of the present inventionmay take the form of an entirely hardware embodiment, an entirelysoftware embodiment (including firmware, resident software, micro-code,etc.) or an embodiment combining software and hardware aspects that mayall generally be referred to herein as a “circuit,” “module” or“system.” Furthermore, aspects of the present invention may take theform of a computer program product embodied in one or more computerreadable medium(s) having computer readable program code embodiedthereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

Embodiments of the invention may be provided to end users through acloud computing infrastructure. Cloud computing generally refers to theprovision of scalable computing resources as a service over a network.More formally, cloud computing may be defined as a computing capabilitythat provides an abstraction between the computing resource and itsunderlying technical architecture (e.g., servers, storage, networks),enabling convenient, on-demand network access to a shared pool ofconfigurable computing resources that can be rapidly provisioned andreleased with minimal management effort or service provider interaction.Thus, cloud computing allows a user to access virtual computingresources (e.g., storage, data, applications, and even completevirtualized computing systems) in “the cloud,” without regard for theunderlying physical systems (or locations of those systems) used toprovide the computing resources.

Typically, cloud computing resources are provided to a user on apay-per-use basis, where users are charged only for the computingresources actually used (e.g. an amount of storage space consumed by auser or a number of virtualized systems instantiated by the user). Auser can access any of the resources that reside in the cloud at anytime, and from anywhere across the Internet. In context of the presentinvention, a user may access environmental illumination data availablein the cloud. For example, a augmented reality component 110 couldexecute on an augmented reality device 100 operated by a user andcollect environment illumination data pertaining to the user's currentenvironment. In such a case, the augmented reality component 110 couldtransmit the collected data to a computing system in the cloud forstorage. When the user again returns to same environment, the augmentedreality component 110 could query the computer system in the cloud toretrieve the environmental illumination data and could then use theretrieved data to realistically model lighting effects on paintedobjects within an augmented reality scene displayed on the augmentedreality device 100. Doing so allows a user to access this informationfrom any device or computer system attached to a network connected tothe cloud (e.g., the Internet).

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. Each block of the block diagrams and/orflowchart illustration, and combinations of blocks in the block diagramsand/or flowchart illustration, can be implemented by special-purposehardware-based systems that perform the specified functions or acts, orcombinations of special purpose hardware and computer instructions.

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

What is claimed is:
 1. A method of displaying a first physical object,comprising: capturing a visual scene for display, wherein the visualscene includes the first physical object and wherein the visual scene iscaptured using one or more camera devices; classifying the firstphysical object as a first predetermined object type, based on one ormore object identifiers associated with the first physical object; uponclassifying the first physical object as the first predetermined objecttype, retrieving predefined geometric information corresponding to thefirst predetermined object type, wherein the predefined geometricinformation provides a three-dimensional model describing a shape ofobjects of the first predetermined object type, and wherein thepredefined geometric information specifies a first region of thethree-dimensional model and one or more augmentations applicable to thefirst region of the objects of the first predetermined object type;determining which one of a plurality of configurations the firstphysical object is currently in, wherein the first physical object isconfigured to be physically manipulatable into each of the plurality ofconfigurations; and rendering a sequence of frames for display in whichthe first region of the first physical object within the captured visualscene is animated in a predefined manner depicting a virtual lightsource within the first physical object, based on the determinedconfiguration of the first physical object, by applying the one or moreaugmentations to a first virtual object generated based on the retrievedpredefined geometric information corresponding to the first physicalobject in the captured visual scene.
 2. The method of claim 1, whereinidentifying the first physical object as the first predetermined objecttype further comprises: analyzing the visual scene to determine aplurality of edges of the first physical object within the visual scene,wherein the first predetermined object type is identified further basedon the determined plurality of edges.
 3. The method of claim 1, furthercomprising: measuring one or more environmental illuminationcharacteristics of an environment in which an augmented reality deviceis located, wherein the environmental illumination characteristicsinclude at least one of a position of a light source within anenvironment in which the augmented reality device is located, an angleof the light source, an indication of whether the light source isomnidirectional, a color of the light source, an intensity of the lightsource and a reflectivity value of the first physical object; andadjusting the appearance of the augmented first physical object based onthe measured one or more environmental illumination characteristics. 4.The method of claim 1, wherein the visual scene is augmented to includeone or more virtual objects, and further comprising: determining one ormore illumination effects associated with a first one of the one or morevirtual objects, wherein the visual scene is augmented further based onthe determined one or more illumination effects.
 5. The method of claim1, wherein identifying the first physical object as a firstpredetermined object type, based on one or more object identifiersassociated with the first physical object, further comprises: receivinga signal associated with the first physical object; and processing thereceived signal to determine the one or more object identifiers that areencoded within the signal.
 6. The method of claim 1, wherein identifyingthe first physical object as a first predetermined object type, based onone or more object identifiers associated with the first physicalobject, further comprises: detecting an embedded marker within the firstphysical object; and determine the first physical object corresponds tothe first predetermined object type, based on the detected embeddedmarker.
 7. The method of claim 6, wherein the embedded marker isembedded in the first physical object using an infrared-absorbingmaterial, and wherein the embedded marker is detected using an infraredcamera coupled to the augmented reality device.
 8. The method of claim1, further comprising: inserting an animated virtual character into thevisual scene, wherein the animated virtual character is depicted in therendered sequence of frames as interacting with the first physicalobject in the rendered sequence of frames, based on the retrievedpredefined geometric information.
 9. The method of claim 8, furthercomprising: determining one or more dynamics rules based on the firstphysical object, wherein the one or more dynamics rules govern thephysics that apply to the animated virtual character; and affecting theinteraction between the animated virtual character and the firstphysical object based on the determined dynamics rules.
 10. Acomputer-readable memory containing a program that, when executed,performs an operation for displaying a first physical object,comprising: capturing a visual scene for display, wherein the visualscene includes the first physical object and wherein the visual scene iscaptured using one or more camera devices; classifying the firstphysical object as a first predetermined object type, based on one ormore object identifiers associated with the first physical object; uponclassifying the first physical object as the first predetermined objecttype, retrieving predefined geometric information corresponding to thefirst predetermined object type, wherein the predefined geometricinformation provides a three-dimensional model describing a shape ofobjects of the first predetermined object type, and wherein thepredefined geometric information specifies a first region of thethree-dimensional model and one or more augmentations applicable to thefirst region of the objects of the first predetermined object type;determining which one of a plurality of configurations the firstphysical object is currently in, wherein the first physical object iscapable of being physically manipulated into each of a plurality ofconfigurations; and rendering a sequence of frames for display in whichthe first region of the first physical object within the captured visualscene is animated in a predefined manner depicting a virtual lightsource within the first physical object, based on the determinedconfiguration of the first physical object, by applying the one or moreaugmentations to a first virtual object generated based on the retrievedpredefined geometric information corresponding to the first physicalobject in the captured visual scene.
 11. The computer-readable memory ofclaim 10, wherein identifying the first physical object as the firstpredetermined object type further comprises: analyzing the visual sceneto determine a plurality of edges of the first physical object withinthe visual scene, wherein the first predetermined object type isidentified further based on the determined plurality of edges.
 12. Thecomputer-readable memory of claim 10, the operation further comprising:measuring one or more environmental illumination characteristics of anenvironment in which an augmented reality device is located, wherein theenvironmental illumination characteristics include at least one of aposition of a light source within an environment in which the augmentedreality device is located, an angle of the light source, an indicationof whether the light source is omnidirectional, a color of the lightsource, an intensity of the light source and a reflectivity value of thefirst physical object; and adjusting the appearance of the augmentedfirst physical object based on the measured one or more environmentalillumination characteristics.
 13. The computer-readable memory of claim10, wherein the visual scene is augmented to include one or more virtualobjects, and the operation further comprising: determining one or moreillumination effects associated with a first one of the one or morevirtual objects, wherein the visual scene is augmented further based onthe determined one or more illumination effects.
 14. Thecomputer-readable memory of claim 10, wherein identifying the firstphysical object as a first predetermined object type, based on one ormore object identifiers associated with the first physical object,further comprises: receiving a signal associated with the first physicalobject; and processing the received signal to determine the one or moreobject identifiers that are encoded within the signal.
 15. Thecomputer-readable memory of claim 10, wherein identifying the firstphysical object as a first predetermined object type, based on one ormore object identifiers associated with the first physical object,further comprises: detecting an embedded marker within the firstphysical object; and determine the first physical object corresponds tothe first predetermined object type, based on the detected embeddedmarker.
 16. The computer-readable memory of claim 15, wherein theembedded marker is embedded in the first physical object using aninfrared-absorbing material, and wherein the embedded marker is detectedusing an infrared camera coupled to the augmented reality device. 17.The computer-readable memory of claim 10, the operation furthercomprising: inserting an animated virtual character into the visualscene, wherein the animated virtual character is depicted in therendered sequence of frames as interacting with the first physicalobject in the rendered sequence of frames, based on the retrievedpredefined geometric information.
 18. The computer-readable memory ofclaim 17, further comprising: determining one or more dynamics rulesbased on the first physical object, wherein the one or more dynamicsrules govern the physics that apply to the animated virtual character;and affecting the interaction between the animated virtual character andthe first physical object based on the determined dynamics rules.
 19. Anaugmented reality device, comprising: a processor; and a memorycontaining a program that, when executed by the processor, performs anoperation for displaying a first physical object, comprising: capturinga visual scene for display, wherein the visual scene includes the firstphysical object and wherein the visual scene is captured using one ormore camera devices; classifying the first physical object as a firstpredetermined object type, based on one or more object identifiersassociated with the first physical object; upon classifying the firstphysical object as the first predetermined object type, retrievingpredefined geometric information corresponding to the firstpredetermined object type, wherein the predefined geometric informationprovides a three-dimensional model describing a shape of objects of thefirst predetermined object type, and wherein the predefined geometricinformation specifies a first region of the three-dimensional model andone or more augmentations applicable to the first region of the objectsof the first predetermined object type; determining which one of aplurality of configurations the first physical object is currently in,wherein the first physical object is capable of being physicallymanipulated into each of a plurality of configurations; and rendering asequence of frames for display in which the first region of the firstphysical object within the captured visual scene is animated in apredefined manner depicting a virtual light source within the firstphysical object, based on the determined configuration of the firstphysical object, by applying the one or more augmentations to a firstvirtual object generated based on the retrieved predefined geometricinformation corresponding to the first physical object in the capturedvisual scene.
 20. The augmented reality device of claim 19, whereinidentifying the first physical object as the first predetermined objecttype further comprises: analyzing the visual scene to determine aplurality of edges of the first physical object within the visual scene,wherein the first predetermined object type is identified further basedon the determined plurality of edges.
 21. The augmented reality deviceof claim 19, the operation further comprising: measuring one or moreenvironmental illumination characteristics of an environment in whichthe augmented reality device is located, wherein the environmentalillumination characteristics include at least one of a position of alight source within an environment in which the augmented reality deviceis located, an angle of the light source, an indication of whether thelight source is omnidirectional, a color of the light source, anintensity of the light source and a reflectivity value of the firstphysical object; and adjusting the appearance of the augmented firstphysical object based on the measured one or more environmentalillumination characteristics.
 22. The augmented reality device of claim19, wherein the visual scene is augmented to include one or more virtualobjects, and the operation further comprising: determining one or moreillumination effects associated with a first one of the one or morevirtual objects, wherein the visual scene is augmented further based onthe determined one or more illumination effects.
 23. The augmentedreality device of claim 19, wherein identifying the first physicalobject as a first predetermined object type, based on one or more objectidentifiers associated with the first physical object, furthercomprises: receiving a signal associated with the first physical object;and processing the received signal to determine the one or more objectidentifiers that are encoded within the signal.
 24. The augmentedreality device of claim 19, wherein identifying the first physicalobject as a first predetermined object type, based on one or more objectidentifiers associated with the first physical object, furthercomprises: detecting an embedded marker within the first physicalobject; and determine the first physical object corresponds to the firstpredetermined object type, based on the detected embedded marker. 25.The augmented reality device of claim 24, wherein the embedded marker isembedded in the first physical object using an infrared-absorbingmaterial, and wherein the embedded marker is detected using an infraredcamera coupled to the augmented reality device.
 26. The augmentedreality device of claim 19, the operation further comprising: insertingan animated virtual character into the visual scene, wherein theanimated virtual character is depicted in the rendered sequence offrames as interacting with the first physical object in the renderedsequence of frames, based on the retrieved predefined geometricinformation.
 27. The augmented reality device of claim 26, furthercomprising: determining one or more dynamics rules based on the firstphysical object, wherein the one or more dynamics rules govern thephysics that apply to the animated character; and affecting theinteraction between the animated virtual character and the firstphysical object based on the determined dynamics rules.